A JavaScript/Node.JS editor/IDE for Mobile phones

This is a blog series about creating a web based editor for JavaScript/Node.JS and web development.
Read part 1:, part 2, part 3:, part 4, part 5 (Swedish), or part 6:

The tiny screens of mobile phones creates a huge design challenge for mobile editors/IDE's.

Coding on a mobile

Writing code on a mobile phone kinda sucks. But why?

When I started with programming we had a 16 MHz computer with a 800x600 resolution display.
And "back in the days" programmers didn't even have a monitor, they had to print the code on paper to see it.
Mobile phones today are 1000 times more powerful then my old 386 PC, and the screens have full HD display (1920 x 1080 pixels) - or better!
The only thing a mobile phone lacks is a proper keyboard.

Text input on mobiles

When buying my very first Smartphone I made sure it had a fold-able keyboard. I really hated typing using the numeric keyboard on my then current "dumb-phone".
The phone I bought also came with a stylus, and surprisingly I ended up using the stylus to type on a on-screen virtual keyboard instead of using the fold-able hardware keyboard!

The fastest input method when it comes to Mobile phone virtual keyboards is a "swype" technique, where instead of clicking on the screen you swipe between letters, and the keyboard figures out the word you want to type.
This works well for long words and common sentences. I do not think it would work as great for writing code though.

There are some cool technologies like laser keyboards, where the phone projects a virtual keyboard on any flat surface using a small laser projector.
The ones I looked at however wasn't great, resulting in lots of unregistered/wrong keys.



If I'm going to buy I new phone, and use it for writing code, I will certainly go with a phone that has stylus support.
There are however not many phones with a stylus on the market today. There's Samsung Galaxy Note, but it's rather expensive.
What special about phones with stylus support is that they have an extra layer to the screen, which allows much better click precision with the stylus, compared to using your fingers.

Making virtual keyboard for the editor

In my quest to improve the develping experience using mobile phones to write code, I decided to build in a virtual keyboard into the editor/IDE. Having a keyboard integrated with the editor have some advantages, for example, plugins can now add special functionality to the alt-keys. For example Alt->S to save. etc. This is good for function discovery. Having the special functionality printed out on the keys make them easier to find!

Keyboard shortcuts & discoverability

With a standard hardware keyboard you can use Ctrl or Alt to make the editor do special things.
But to my surprise, many users are not familiar with the concept of keyboard shortcuts!! Most users don't even use Ctrl+C to copy!
This is also a reason why software and developer tools are hard to sell, you need to show users how to use it.

Currently the editor has a "context" menu that comes up when you right click. That is however not enough for discoverability!
Discoverability in user interfaces and software in general is a very hard problem!

Another reason for making my own virtual keyboard is that I can choose exactly what buttons to include and where to put them ...

I first started with naively designing a keyboard that looked like a real keyboard. Real hardware keyboards however has too many keys to fit on a mobile phone screen without the keys becoming too small.

Then I got an idea to use alt-buttons, so that when a alt-button is pressed, the available keys change functionality, kinda like with a real keyboard where Shift+1 becomes ! or AltGr+2 becomes @. (you will get a different character depending on your keyboard)
I made one row with numbers and 3 rows with A-Z latin characters, and also squeezed space, enter, and two alt buttons in there. To make a backspace you had to press alt+enter.
I ended up with a keyboard with relative big buttons, but coding on it sucked, having to click two times just to insert a comma or punctuation was too slow! So it was back to the "drawing board"

For the next iteration I was more scientific. I analyzed all source files in the editor's code base, including third party modules. And counted how common each character was. It was no surprise that besides letters A-Z, space/tabs and Enter was common, but dot and comma was also very common, and all those buttons needed to be easily accessed.
I placed the most important buttons in the middle, and made them a bit bigger then the rest. I had to split A-Z into two groups to achieve this, but it ended up rather nicely, and is OK to code with.

Performance optimizations

I have a almost ten year old Nokia phone for performance reference.
The Nokia web browser is a bit slow, so I mostly used Opera Mobile on it.
My reasoning for using an old Nokia for reference is that, if I can make the experience smooth on my old Nokia, I figure it will also be smooth on pretty much all current smartphones.

For the very first version of the vertical keyboard I used HTML buttons. They where snappy on my developer machine and mobile emulator, but the experience was very bad on my old Nokia. Not only was the buttons really small and hard to hit. The input was also laggy.

The browser's graphics engine

CSS is very nice to work with, but can be a bit tricky when you want everything to look pixel perfect on all browsers and screen resolutions.
My editor currently do all text rendering using the Canvas element! Which means I also have to implement everything from scratch, things like text selection, etc.
The reason why I choose Canvas for the text rendering, instead of the browser's text rendering that is much faster, is so that I do not have to fight the DOM and "contenteditable". I still use the DOM and "contentedibale" for previewing, but "contentedibale" has lots of incompatibility between browsers. So I do not regret using the Canvas for the main text rendering.

I think both Chrome and Firefox use the Skia graphics engine for the Canvas.
It would be very cool if someone could use v8 (Chrome's JavaScript engine) and Skia to make a framework for developing cross platform apps, kinda like Flutter works, (which also use the Skia graphics engine)
It would be nice for writing high performance graphical applications using JavaScript. And as my editor mostly uses the Canvas, it would be easy to port to such a framework.

Rendering optimizations

To fix the laggy input I started picking the lowest hanging fruits optimization wise, and started with the rendering. I saw a huge improvement by only re-rendering the row that was being edited.

I also shaved off a couple of millisecond by using a {object} instead of a Class object with prototype when copying the grid buffer into the rendering pipeline.
The reason why I'm giving the render-functions a copy of the buffer and not the buffer as is, is because then renders doesn't have to clean up afterward (like resetting colors etc). The copied buffer can just be thrown away when the rendering is done.

After much testing I also found out that having a HTML keyboard on top of a canvas element slowed down the rendering significantly on Opera Mobile.
So I went with a canvas based virtual keyboard instead. The nice thing about the canvas is that I can control everything down to the sub-pixel. And can thus make sure there are not a single pixel wasted. And that the keyboard is always rendered correctly on all screen sizes. With the HTML keyboard I needed to have a bunch of @media (max-width) in the CSS.

Another thing that was slow on my old Nokia was scrolling. I made it faster by simply moving the image up when scrolling down, and only render the missing lines.

Optimizations are evil

When doing optimizations the code often get harder to understand, and you sometimes get weird edge cases. For example when scrolling in the editor using a touch-screen, you get a helping line to show where the scroll-zone is, you can only scroll by swiping inside that zone. Swiping outside that zone makes a text selection instead. But as my scrolling optimization makes a copy of the frame, and render it above or below, that helper line also get copied, so I had to fill that line with the background, and also make sure the line was not drawn on-top of text.

People tend to think that optimization, using for example a more complicated algoritm to solve a problem is a no-brainer, eg. always go for the faster algoritm. But there are always trade-offs! A more complicated algoritm is harder to understand and thus harder to implement and probably more prone to bugs and edge cases. And optimizations in general are often mere hacks - that will easily break, or you get edge cases where those hacks cause weird issues.

People often complain about how computers get faster and faster, while software gets slower and slower. Older programs, written in Assembly or C are certainly faster then "modern" web applications, but the web app is probably easier to maintain, and you can add more features - faster.

It however is possible to make that web app run fast, even faster then non-optimized C.
When optimizing JavaScript I often get 10x-100x returns, eg the functions gets 10-100 times faster. But you shouldn’t optimize everything due to opportunity cost and maintenance cost. And over-complicated code will often end up running slower then the same function naively implemented! So you always need to measure! And only optimize where it's needed!

Developer machines vs user hardware

Another problem is that developers (like myself) are often on machines that are 10 times faster then ordinary devices. I do most on my developing on my PC that also serves as a gaming rig, it's not the fastest machine out there, but it's certainly faster then a Macbook pro, which in turn is much faster then third world mobile devices

So while the web app runs smooth on my developer machine, it can be laggy on user's cheap smartphones. So while you should never prematurely optimize, you should always test your apps on old devices! And don't forget to limit the network throughput to simulate a shitty Internet connection.

Problems designing for a small screen

The more I code using my mobile phone, the more issues I find ...

Fighting the built in virtual keyboard

Whenever you click on a textarea or input, the mobile's "native" keyboard will show up, and likely set zoom to a ridiculous level.
I will probably end up "faking" all text input elements to prevent the "native" keyboard, so users can use the editor's keyboard instead.

Widgets and plugin

The editor has a plugin system where you can create "widgets" that are built using HTML DOM elements. These work very nice on a big screen computer, but they are hard to use on mobile and cover up a lot of screen real-estate, especially if zoomed in.
I've been thinking of making all elements canvas based. But HTML elements are already optimized and fast enough. Most HTML elements are also "native" which means they are provided by the operating system (OS) graphics user interface (GUI), meaning they have been designed to work optimally for the device.
Making everything Canvas based would be a lot of work, and with little benefit. CSS is also very nice to work with, compared to drawing stuff yourself.

Mobile screen flipping

On mobiles it's easy to flip the screen, wide screen is usually very wide, and if you turn it vertically it becomes very high. I first used a separate keyboard layout for horizontal and vertical mode, but as I like the vertical much more, I now also use it while in horizontal.

Copy & Paste

Copy/pasting and text selecting works different on Mobile phones, compared to using a mouse on a PC, or touch pad on a laptop.
Pasting on most mobile devices is done by long pressing until you get the copy option. But you only get that option on text elements or input! So because my editor use the canvas to render text - I have to re-implement things like copy/pasting and text selection from scratch.

Browser are also very conservative when it comes to accessing the clipboard, and for good reason as the clipboard can contain passwords and other sensitive data.
Modern browsers have a clipboard API, but for old browsers the text editor have to show a textarea to manually copy/pate from.


Written by December 12, 2018.


Follow me via RSS:   RSS https://zäta.com/rss_en.xml (copy to feed-reader)
or Github:   Github https://github.com/Z3TA